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1.0  INTRODUCTION 


As  part  of  Its  effort  to  develop  external  physical  security  sys- 
tems for  the  Department  of  Defense,  the  Electronic  Systems  Division 
(ESD)  of  the  United  States  Air  Force,  under  its  Base  and  Installa- 
tion Security  System  (BISS)  program,  acquired  advanced  development 
models  of  three  automated  identity  verification  systems  for  use  in 
entry  control. 

The  identity  verification  systems  are  designed  to  provide 
verification  of  the  claimed  identity  of  persons  entering  or  leaving 
a restricted  area  through  an  Entry  Control  Point.  The  systems 
accomplish  this  verification  by  examining  unique  and  measurable 
properties  of  a person's  speech,  handwriting,  and  fingerprint.  The 
three  systems  that  were  evaluated  include  the  Automatic  Speaker 
Verification  (ASV)  system,  the  Automatic  Handwriting  Verification 
(AHV)  system  and  the  Automatic  Fingerprint  Verification  (AFV)  system. 
The  MITRE  Corporation  was  given  the  responsibility  for  installing, 
testing  and  evaluating  the  three  systems/techniques  both  at  MITRE 
and  at  a field  test  site.  This  volume,  Volume  IV,  of  the  test  report 
will  address  in  detail  the  tests,  results,  and  evaluations  of  the  AFV 
system  as  briefly  discussed  in  Volume  I,  the  Executive  Summary. 

The  Automatic  Fingerprint  Verification  system  was  acquired  from 
Calspan  Corporation  of  Buffalo,  N.Y.,  by  ESD  in  the  Spring  of  1976. 
The  AFV  system  was  installed  in  the  Entry  Control  Laboratory  at  The 
MITRE  Corporation  in  Bedford,  Massachusetts  and  tested  and  evaluated 
during  Phase  II  of  the  test  program.  After  completion  of  Phase  II, 
the  AFV  system  was  taken  to  Pease  AFB,  Portsmouth,  New  Hampshire, 
for  evaluation  in  a field  environment.  In  both  test  phases,  the 
AFV  system  was  tested  alongside,  and  in  conjunction  with,  the  AHV 
and  ASV  systems  described  in  Volumes  II  and  III  of  this  report. 
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2.0  SUMMARY 


A summary  of  the  Phase  II  and  Field  Test  data  is  presented  In 
Table  I.  The  AFV  system  was  not  available  for  testing  during  Phase  I 
as  were  the  ASV  and  AHV  systems.  Phase  II  was  divided  into  two  test 
periods.  The  results  of  the  first  test  period  are  not  included  in 
the  summary  because,  as  will  be  reported,  a parameter  of  the  AFV 
algorithm  was  in  slight  error  and  required  modification.  Therefore, 
only  the  results  of  the  second  test  period  of  Phase  II  are  included 
in  Table  I.  There  was  no  change  in  the  algorithm  between  Period  2 
of  Phase  II  and  the  Field  Test.  Because  of  the  very  small  number  of 
females  participating  in  the  Field  Test,  only  the  error  rates  for  the 
male  population  are  included  for  comparison  in  the  summary. 


Table  I 

Summary  of  AFV  Test  Results 


ij  Phase  II 

Field 

Enrollments 

Male 

166 

268 

Female 

34 

7 

Participants 

Male 

162 

202 

Female 

34 

6 

Type  I Error  Rate  (Male) 

4.58% 

6.54% 

Type  II  Error  Rate  (Male) 

2.18* 

2.32% 

Verification  Time  (seconds) 

8.53 

8.94 

10 


Enrollments  are  the  number  of  people  enrolled  in  the  AFV  system 
for  Phase  II  and  the  Field  Test.  Participants  are  the  number 
of  people  that  used  the  AFV  system  at  least  once  after  being  enrolled 
in  the  system. 

The  Type  I error  rate  is  the  percentage  of  users  who  were  falsely 
rejected  by  the  system.  That  is,  they  were  who  they  claimed  to  be 
but  they  could  not  pass  the  AFV  system.  The  BISS  requirement  of  a 
Type  I error  rate  of  1%  was  not  achieved  by  the  AFV  system. 

The  Type  II  error  rate  is  the  percentage  of  users  who  could  have 
been  accepted  by  the  AFV  system  even  though  they  were  not  who  they 
claimed  to  be.  This  data  was  obtained  by  computer  comparisons  of 
randomly  selected  fingerprints,  collected  in  real  time,  with  the 
reference  file  library.  The  BISS  requirement  of  a Type  II  error 
rate  of  2%  was  not  achieved  by  the  AFV  system. 

The  verification  time  is  the  average  time  from  when  an  individ- 
ual's identification  number  is  accepted  until  a decision  is  made 
for  that  person  by  the  AFV  system.  To  provide  the  BISS  required 
throughput  using  a detention  module,  the  verification  time  could  not 
exceed  six  seconds.  This  time  was  not  achieved  by  the  AFV  system. 

In  Phase  II,  the  Type  I and  Type  II  error  rates  were  higher  for 
females  than  for  males.  In  the  Field  Test,  no  comparison  was  made 
because  only  seven  females  were  enrolled  on  the  system. 

In  both  tests,  the  Type  I error  rate  was  higher  before  noon  than 
it  was  in  the  afternoon.  It  has  been  suggested  that  this  occurs 
because  the  hand  is  relatively  dry  and  oil-free  in  the  morning. 
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The  Type  I performance  for  the  Primary  finger  was  much  better 
(sj14%)  than  it  was  for  the  Alternate  finger  (ss43X),  and  it  was 
much  better  for  the  Primary  matcher  (~15X)  than  it  was  for  the 
Secondary  matcher  (~53X). 

A large  percent  of  the  enrolled  population  had  difficulty  veri- 
fying in  spite  of  reenrollments.  About  22X  of  the  users  had  a Type 
I error  rate  greater  than  5X.  Moat  of  this  was  due  to  poor  quality 
fingerprints. 

The  AFV  system  was  awkward  for  several  users.  The  scanner  is 
not  designed  for  those  with  very  large  or  very  small  fingers.  A 
few  users  with  arthritis  or  unusual  bone  structure  had  difficulty 
placing  their  fingers  in  the  scanner. 

Sophisticated  Type  II  spoofing  tests  were  conducted  against  the 
AFV  system.  The  results  are  presented  in  another  document. 


3.0  OBJECTIVES 


The  objectives  of  Phase  II  and  the  Field  Test  for  the  AFV  system 
were  the  same.  The  objectives  that  will  be  discussed  in  Section  6.0 
are  listed  below  by  number,  title  and  paragraph. 


No.  Objective  Paragraph 

23  Type  I Error  Analysis  in  Real  Time  6.1 

24  Type  I Error  Analysis  in  Non-Real  Time  6.2 

25  Independence  of  Type  1 Scores  6.3 

26  Type  II  Error  Analysis  in  Real  Time  6.4 

27  Type  II  Error  Analysis  in  Non-Real  Time  6.5 

28  Independence  of  Type  II  Scores  6.6 

29  Sensitivity  Analysis  of  Type  I and  6.7 

Type  II  Errors  to  Thresholds 

30  Verification  Time  Analysis  6.8 


A detailed  discussion  of  these  objectives  including  the  test 
approach,  data  required,  data  reduction,  and  data  analysis  are  pre- 
sented in  other  documents.  The  objectives  pertaining  to  human  factors, 
enrollment  procedure,  equipment  reliability,  environment,  and  hybrid 
systems  are  discussed  in  Volume  V. 
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4.0  SYSTEM  DESCRIPTION 


4.1  EQUIPMENT 

The  AFV  system  Is  composed  of  three  basic  components:  an  Entrant 
Terminal,  an  Operator  Terminal,  and  a Central  Station.  A block 
diagram  of  the  AFV  system  is  shown  in  Figure  1. 

4.1.1  Entrant  Terminal 

The  Entrant  Terminal  is  shown  in  Figure  2.  This  terminal  con- 
tains a keypad  for  entering  the  entrant's  Identification  number,  a 
display  for  providing  instructions  and  entry  status  to  the  user,  and 
a mechanical-electro-optical  sensor  that  scans  the  fingerprint  and 
converts  the  image  into  digital  data.  Mounted  on  top  of  the  entrant 
terminal  is  a finger  position  display  for  assisting  the  user  in  plac- 
ing his  finger  correctly  in  the  scanner. 

4.1.2  Operator  Terminal 

The  Operator  Terminal  contains  a similar  keypad  to  the  Entrant 
Terminal,  functional  pushbuttons  for  enrolling  and  general  control  of 
the  system's  operation,  and  an  alphanumeric  display  for  displaying 
system  status,  enrollment  instructions,  alarm  conditions  and  user 
verification  status. 

4.1.3  Central  Station 

The  Central  Station  contains  a preprocessor,  the  central  proces- 
sing unit  (CPU)  (a  Texas  Instrument  980B  minicomputer)  and  a disc 
storage  unit  for  storing  the  application  program  and  the  library  of 
user  fingerprint  reference  files  established  during  the  enrollment 


entrant  central  operator 


Figure  1.  Automatic  Fingerprint  Verification  System. 


process.  The  AFV  system  was  connected  to  the  CPU  of  the  ASV  system 
which  was  employed  as  an  Integration  Processor  (IP)  that  coordinated 
the  operation  of  all  three  verification  systems.  The  CPU  forwarded 
to  the  Integration  Processor,  via  the  IP  Interface,  appropriate 
data  to  be  recorded  on  magnetic  tape  for  use  In  determining  error 
rates  and  for  the  printout  of  AFV  verification  data. 

4.2  OPERATION 

The  following  provides  a brief  description  of  the  operation 
of  the  AFV  system. 

The  scanner  mechanism,  located  under  where  the  finger  is  placed 
on  the  user  terminal,  contains  a light  source,  prism,  photodiode 
detector,  and  associated  mirrors  and  drives.  A 3/8  inch  by  3/8  inch 
binary  image  of  the  fingerprint  area  is  transmitted  to  the  preprocessor. 
The  preprocessor  scans  the  data  for  ridge  endings  and  ridge  branchings 
collectively  called  minutiae.  A typical  fingerprint  image  is  shown 
in  Figure  3.  The  fingerprint  data  is  comprised  of  a collection  of 
unit  vectors  which  show  the  relative  position  and  orientation  of 
the  minutiae.  This  set  of  minutiae  position  (X^Y^)  and  orientation 
(0^)  coordinates  is  compared  with  coordinates  of  an  individual's 
minutiae  data  obtained  during  enrollment  and  on  file  in  the  Central 
Station.  Figure  4 shows  many  of  the  unit  vectors  for  the  fingerprint 
of  Figure  3. 

In  correlating  a person's  minutiae  data  with  the  data  stored  in 
the  reference  file,  a score  is  produced  indicating  the  degree  of  cor- 
relation. A pass/fall  decision  is  made  by  comparing  the  score  with  a 
predetermined  threshold.  There  are  two  algorithms  used  for  determin- 
ing a pass/fail  decision.  The  first  algorithm  is  referred  to  as  the 
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Figure  3.  Fingerprint  Image 


Figure  4.  Relative  Position  and  Orientation 
of  Minutiae. 
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Primary  matcher  and  the  second  algorithm  referred  to  an  the  Secondary 
matcher.  The  choice  as  to  which  algorithm  is  used  is  made  at  the 
time  of  enrollment  and  is  determined  by  the  performance  of  the  two 
matchers  with  a particular  finger  image.  The  Primary  matcher  is  the 
preferred  matcher  and  is  predominantly  used.  The  algorithm  that  is 
chosen  is  permanently  assigned  to  the  individual. 

4.2.1  Enrollment 

Upon  arriving  at  a test  area,  a test  participant  was  informed 
by  an  audiovisual  presentation  about  the  test  program  and  enroll- 
ment and  verification  procedures.  The  enrollee  then  provided,  via 
an  enrollment  card,  personnel  data  required  for  the  verification 
process  and  data  analysis. 

To  enroll  in  the  AFV  system,  the  enrollee  must  have  first  en- 
rolled in  the  ASV  system.  This  was  required  because  of  the  way  in  which 
the  AFV  system  was  integrated  into  the  IP  configuration  for  testing, 
data  recording  and  generation  of  statistics.  The  personal  data  en- 
tered into  the  CPU  of  the  ASV  system  was  made  available  to  the  AFV 
system  for  the  recording  of  AFV  data  by  the  IP. 

The  operator  began  AFV  enrollment  by  entering  the  enrollee' s 
identity  number  via  the  keypad  on  the  Operator  Terminal  thus 
creating  a file  in  the  AFV  computer.  Next,  the  enrollee  was  instructed 
to  place  each  of  his  fingers  in  the  scanner  (on  the  Entrant  Terminal) 
where  a fingerprint  image  is  obtained.  After  all  eight  fingers  were 
observed,  the  computer  chose  the  fingerprint  that  best  met  the  require- 
ments through  a Quality  Index  based  primarily  on  the  clarity  of  the 
print  and  the  number  of  minutiae  found.  The  enrollee  then  placed  that 
finger,  known  as  the  Primary  finger,  in  the  scanner  where  it  was  held 
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stationary  for  four  scans.  After  the  fourth  scan,  the  enrollee  re- 
moved his  finger  and  then  placed  it  back  In  the  scanner.  This  pro- 
cedure was  repeated  for  the  next  six  scans  making  a total  of  ten 
fingerprint  Images  available  for  generating  a reference  file  against 
which  all  future  prints  would  be  compared.  Upon  completing  the  en- 
rollment for  the  Primary  finger,  a second  finger  was  chosen  by  the 
AFV  computer  as  an  Alternate  finger  and  a reference  file  generated 
in  the  same  manner  as  for  the  Primary  finger. 

After  the  reference  files  were  established  for  both  Primary  and 
Alternate  fingers,  the  enrollment  process  was  complete  and  the  enrollee 
was  given  a FINGERSCAN  user  card  to  remind  him  which  finger  was  his 
Primary  and  which  was  his  Alternate. 

It  was  not  uncommon  to  repeat  the  enrollment  process  either 
because  the  enrollee  used  the  wrong  finger,  had  difficulty  using 
the  terminal,  or  the  computer,  because  of  poor  matcher  scores,  would 
suggest  reenrollment.  A normal  enrollment  took  about  3.0  minutes. 

4.2.2  Verification 

The  verification  process  required  the  test  participant  to  enter 
his  identification  number  and  place  his  Primary  finger  in  the  scanner. 
If  the  comparison  of  the  real  time  fingerprint  data  with  the  user's 
data  in  the  reference  file  satisfied  a threshold  level,  the  user 
was  verified  and  got  a THANK  YOU  message  on  the  message  display  of 
the  entrant  terminal.  If  the  data  comparison  did  not  pass  the  thres- 
hold level,  a message  TRY  FINGER  AGAIN  was  displayed  and  the  user 
reinserted  his  Primary  finger.  When  the  Primary  finger  failed  three 
scans,  the  message  TRY  ALTERNATE  FINGER  was  displayed.  Again  the 
individual  was  given  up  to  three  scans  to  verify  using  his  Alternate 
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finger.  Thus,  two  basic  three  scan  strategies  are  combined  to  gen- 
erate a six  scan  decision  strategy  and  the  user  was  not  rejected 
unless  he  failed  on  all  six  scans:  three  on  the  Primary  finger  and 
three  on  the  Alternate  finger.  When  an  authorized  person  failed  to 
verify  on  all  six  scans,  a Type  I error  was  generated  and  the  user 
got  a THANK  YOU  message  on  the  message  display  indicating  that  he 
was  finished.  Detaining  or  rejecting  the  individual  was  not  pa; 
of  the  test  procedure. 
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5.0  TEST  PROGRAMS 


5.1  GENERAL 

The  Automatic  Fingerprint  Verification  system  was  evaluated 
during  two  test  phases  conducted  by  The  MITRE  Corporation.  The 
first  test  phase  took  place  at  MITRE  and  is  known  as  Phase  II.  The 
second  test  phase  evaluated  the  AFV  system  in  a more  realistic  environ- 
ment at  Pease  AFB  in  Portsmouth,  New  Hampshire  and  is  known  as  the 
Field  Test.  A general  description  of  Phase  II  and  the  Field  Test  follows. 

5.2  PHASE  II 

5.2.1  Description 

Phase  II  was  conducted  during  August,  September  and  October  of 
1976  at  the  Entry  Control  Laboratory  of  The  MITRE  Corporation  in 
Bedford,  Massachusetts.  A picture  of  the  user  terminals  as  they 
were  installed  in  the  laboratory  is  presented  in  Figure  5.  The  goals 
of  Phase  II  were  to: 

1.  Evaluate  the  AFV  Type  I and  Type  II  error  rates. 

2.  Observe  user  performance  and  identify  any  learning 
curve  effects. 

3.  Establish  the  degree  of  independence  between  the 
three  identity  verification  systems. 
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Figure 


In  addition,  Phase  II  was  to  satisfy  the  test  objectives  and 
provide  other  related  information  as  detailed  in  another  document. 

Because  of  a parameter  change  in  the  decision  algorithm,  Phase 
II  was  divided  into  two  test  periods.  Period  1 occurred  during  the 
first  three  weeks  of  test  followed  by  Period  2 which  lasted  the  re- 
maining six  weeks.  During  the  first  few  weeks  of  Phase  II,  Type  II 

errors  were  unexpectedly  low,  and  individuals  with  good  quality 
prints  were  found  to  be  causing  Type  I errors.  Calspan  Corporation 
found  the  reason  for  this  to  be  a parameter  of  the  decision  logic 
that  was  set  to  a wrong  value.  For  a pass/fail  decision  to  occur, 
an  Individual  must  register  a minimum  number  of  minutiae  pairs.  For 
Period  1 this  number  was  set  at  six  pairs  when  it  should  have  been 
set  at  three  pairs.  For  Period  2 and  the  Field  Test,  the  parameter 
was  properly  set  to  three.  Except  where  noted,  this  report  will 

discuss  only  the  results  pertinent  to  Period  2 of  Phase  II. 

5.2.2  Enrollment  and  Verification  Process 

The  Phase  II  population  was  composed  of  MITRE  employees  consist- 
ing of  programmers , technicians,  engineers  and  secretaries.  The  test 
participants  would  arrive  at  the  Entry  Control  Laboratory  as  per  an 
enrollment  schedule  in  groups  of  four.  Upon  arrival,  the  enrollees 
were  informed,  via  an  audiovisual  presentation,  about  the  test  pro- 
gram, the  operation  of  the  equipment  and  what  would  be  expected  of 
them  during  the  test  program.  After  completing  an  enrollment  card, 
the  test  participant  was  enrolled  in  the  AFV  system  as  per  A. 2.1. 
After  enrollment,  the  test  participants  were  encouraged  to  return  to 
the  laboratory  twice  daily  to  verify  their  identity  using  the  equip- 
ment as  per  4.2.2.  To  prevent  a user  from  being  discouraged  if  he 


..  . . 


24 


consistently  failed,  the  verification  results  were  not  presented  to 
the  user.  When  a person  failed  to  appear  at  the  laboratory  on  any 
one  day,  the  system  operator  contacted  the  individual  and  encouraged 
participation.  The  Entry  Control  Laboratory  was  open  each  weekday 
from  the  hours  of  0800  to  1630. 

5.2.3  Imposter  Data 

To  establish  a random  Type  II  error  rate  for  the  AFV  system, 
imposter  data  was  gathered  on  every  fourth  verification  attempt  of 
each  test  participant.  The  imposter  data  consisted  of  three  scans 
each  of  the  Primary  and  Alternate  fingers.  Unless  an  individual 
kept  track  of  the  number  of  attempts  they  had  made,  they  had  no 
knowledge  as  to  when  they  would  be  an  imposter.  An  imposter  attempt 
could  not  be  differentiated  from  an  attempt  in  which  the  individual 
failed  to  pass  the  verification  on  the  first  five  scans.  The 
imposter  data  was  used  by  the  computer  during  off  hours  to  generate 
Type  II  error  statistics.  This  was  accomplished  by  comparing  each 
of  the  imposter's  fingerprint  data  with  all  the  reference  file 
fingerprint  data  of  others.  When  a match  occurred,  the  appropriate 
Type  II  error  was  noted. 

5.3  FIELD  TEST 


5.3.1  Description 

The  Field  Test  was  conducted  from  November  1976  to  February  1977 
at  the  Entry  Control  Point  (ECP)  to  the  Weapons  Storage  Area  (WSA)  at 
Pease  AFB,  Portsmouth,  New  Hampshire.  The  MITRE  Corporation  under 
the  direction  of  the  Base  and  Installation  Security  System  Program 
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Office  (BISSPO)  acted  as  the  Responsible  Test  Organization  In  super- 
vising, Installing,  testing  and  evaluating  the  Identity  Verification 
Systems.  The  goals  of  the  Field  Test  Included  those  In  5.2.1  plus 
several  new  goals.  The  new  goals  were  to: 

1.  Determine  performance  In  a realistic  environment  with 
a typical  user  population,  and 

2.  Determine  user  reactions  to  the  techniques. 

Further  information  regarding  the  details  of  the  Field  Site,  overall 
system  set-up  and  other  Information  related  to  the  purpose  and  objectives 
is  provided  in  another  document. 

5.3.2  Enrollment  and  Verification 


All  personnel  with  access  to  the  WSA  were  scheduled  to  be  en- 
rolled in  the  Identity  Verification  Systems.  Of  the  more  than 
400  badged  personnel,  the  most  frequent  users  were  the  people  from 
the  Munitions  Maintenance  Squadron  (MMS)  and  the  509th  Security 
Police  Squadron  (SPS).  Other  less  frequent  users  worked  in  the 
Engineering  and  Supply  Squadrons. 

The  user  population  was  predominantly  male  (6  females)  and  con- 
sisted of  officers,  non-commissioned  officers,  airmen  and  civilians. 
Their  jobs  were  varied  and  Included  mechanics,  security  police,  ad- 
ministrators, electricians  and  general  maintenance  workers. 

The  enrollment  process  for  the  Field  Test  was  similar  to  the 
enrollment  process  of  Phase  II  with  the  exceptions  described  herein. 
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Because  of  the  environment  of  the  user  population  for  the  Field 
Test  was  different  from  the  users  in  Phase  II,  a new  audiovisual 
briefing  was  designed  and  the  enrollment  card  was  changed  slightly 
to  reflect  personal  data  unique  to  a military  environment.  During 
enrollment,  there  were  two  Air  Force  operators  on  duty  in  the  van  and 
one  Air  Force  Operator  on  duty  at  the  briefing  which  was  given  at 
the  guard  building  located  at  the  entrance  to  the  WSA.  The  MITRE 
Test  Engineer  remained  in  the  test  van  to  ensure  the  quality  of  en- 
rollments. 

After  completing  enrollment,  the  test  participants  were  required 
to  verify  on  the  equipment  each  time  they  entered  or  exited  the  WSA. 
The  test  van  was  in  operation  each  weekday  from  0600  hours  to  1630 
hours. 

The  test  participants  were  never  detained  if  the  AFV  system 
failed  to  verify  their  identity.  When  a participant  failed,  his 
Identification  number  and  time  of  failure  on  the  AFV  system  was 
printed  on  a teletypewriter.  In  many  cases,  the  operators  on  duty 
were  able  to  ask  the  participant  why  he  might  have  failed.  The 
failure,  reasons  given  and  operator's  observations  were  entered  in 
a log  book. 

It  should  be  noted  that  the  test  participants  were  required  to 
adhere  to  the  Air  Force's  own  security  procedures.  The  Field  Test 
was  not  meant  to  supplement  or  interfere  with  the  security  procedures 
normal  for  the  WSA. 
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5.3.3  Imposter  Data  (Field  Teat) 


The  Field  Test  Imposter  data  for  Type  II  error  analysis  was 
obtained  as  described  In  5.2.3  except  that  the  data  was  gathered 
on  every  8th  verification  attempt  Instead  of  every  4th  verification 
attempt.  The  reason  for  this  change  was  because  there  was  more 
verification  activity  in  the  Field  Test  with  some  individuals  verify- 
ing up  to  8 times  in  one  day  and  also  because  the  collection  of  the 
Imposter  data  takes  additional  time,  which,  for  a large  population, 
would  cause  queuing  at  the  terminal. 

5.4  PARTICIPATION  OF  TEST  POPULATIONS 

Table  II  presents  the  general  population  statistics  for  Phase  II 
and  the  Field  Test. 


Table  II 


User  Population  Data 


PHASE  II 

FIELD  TEST 

Enrolled 

200 

275 

Male  Users 

162 

202 

Female  Users 

34 

6 

Total  Users 

196 

208 

Of  the  total  enrolled  population,  98%  for  Phase  II  and  76%  for  the 
Field  Test  returned  for  at  least  one  verification.  For  the  Field 
Test  it  had  been  planned  to  enroll  all  450  persons  with  access  to  the 


WSA;  however,  because  of  the  time  of  year  and  work  schedules  many 
people  would  not  be  visiting  the  area  so  those  people  were  not  en- 
rolled. 

5.4.1  Pedestrian  Traffic  Flow 

The  number  of  people  verifying  each  day  fluctuated  considerably 
depending  on  work  schedule  and  proximity  to  holidays,  snowstorms,  etc. 

The  number  of  dally  verifications  Is  shown  in  Figure  6 for 
Phase  II  and  the  Field  Test.  The  beginning  of  each  week  is  shown 
by  a vertical  line.  The  breaks  in  the  graph  represent  those  days 
when  the  AFV  system  was  not  in  operation  due  to  storms  or  holidays. 

Figure  7 shows  the  typical  dally  usage  of  the  AFV  system  for 
Phase  II  and  the  Field  Test.  The  figure  shows  for  Phase  II  a heavy 
usage  that  gradually  declines  during  the  mornings  but  again  picks 
up  in  the  early  afternoon. 

For  the  Field  Test,  there  were  three  periods  of  heavy  usage: 
between  the  hours  of  0640  and  0740,  1120  and  1200,  and  1540  and  1640. 

5.4.2  Frequency  of  AFV  Usage 

To  show  the  frequency  of  AFV  usage,  the  total  number  of  verifica- 
tion attempts  was  divided  into  14  groups  of  ten  verifications  each. 

The  number  of  individuals  contributing  to  the  verifications  in  each 
group  was  then  collated  and  plotted  in  the  histograms  of  Figure  8 
for  Period  2 of  Phase  II  and  the  Field  Test.  Though  there  were  about 
the  same  number  of  users,  the  Field  Test  partlcpants  used  the  AFV 
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Figure  7.  Typical  Daily  Traffic  Flow. 
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system  much  more  frequently.  This,  of  course,  was  due  to  the  fact 
that  the  Field  Test  was  in  operation  for  a longer  period  of  time  and 
the  participants  verified  each  time  they  entered  or  exited  the  WSA. 
Whereas,  in  Phase  II,  the  test  participants  were  not  to  verify  more 
than  twice  daily. 


6.0  TEST  RESULTS 


6.1  OBJECTIVE  23  - TYPE  I ERROR  ANALYSIS  IN  REAL  TIME 


6.1.1  Statement  of  Objective 

To  estimate  the  Type  I error  rate  (failure  to  verify  proper  iden- 
tity when  the  representation  is  actually  true)  and  to  determine  the 
confidence  limits  of  the  Type  I error  estimate. 

6.1.2  Type  I Error  Rate 

Each  time  a person  entered  a valid  identification  number  into 
the  AFV  system  via  the  keypad  on  the  Entrant  Terminal,  a verification 
attempt  was  registered.  A failure  to  pass  an  attempted  verification 
is  classified  as  a Type  I error.  The  number  of  Type  I errors  were 
accumulated  on  a daily  basis  throughout  the  test  periods  and  are  pre- 
sented in  Figures  9 and  10  for  Phase  II  and  the  Field  Test  respec- 
tively. 


The  Type  I error  rate  was  determined  using  the  following  formula: 


Type  I Error  Rate  (X) 


I Errors 10Q 

Verification  Attempts 


The  error  rates  for  the  Field  Test  and  for  Period  2 of  Phase  II 
are  llRted  below  in  Table  III. 
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Table  III 
Type  I Error  Rates 


PHASE  II 

FIELD 

PERIOD  2 

TEST 

ATTEMPTS 

3639 

7152 

TYPE  I ERRORS 

193 

468 

TYPE  I ERROR  RATE 

5.30% 

6.54% 

The  Type  I data  shows  that  the  Field  Test  had  a higher  error 
rate  than  Phase  II  and  submitting  the  data  to  an  F ratio  test* 
shows  the  error  rates  to  be  significantly  different,  i.e.. 


3639  469 

7152  x 194 


= 1.23 


F „ (388,938)  a 1.11 
a=.9 


In  addition  to  a higher  Type  I error,  the  Field  Test  also  shows  a 
definite  increasing  error  rate  as  a function  of  time  which  is  not 
supported  by  Phase  II  (compare  Figures  9 and  10) . It  is  felt  that 
the  reason  for  these  differences  is  because  in  Phase  II  there  were 
many  reenrollments  and  the  test  was  4-1/2  weeks  shorter  with  50% 
fewer  verification  attempts  compared  to  the  Field  Test. 


*If  the  parameter  F is  less  than  the  test  variable  T than  at  the 
90%  confidence  level,  the  two  test  groups  have  significantly 
different  error  rates.  If  F is  greater  than  T then  at  the  90% 
confidence  level,  it  cannot  be  said  that  the  two  test  groups 
have  significantly  different  error  rates.  See  Appendix  A. 
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6.1.2. 1 Confidence  Limits  of  the  Type  I Error  Estimate*.  The 
upper  bound  on  the  Type  I error  at  90%  confidence  la  given  by  the 
Chl-squared  test: 

ErrorsQ  ^ ■ [ *2  (2e*-2)]/2 

where  e ■ observed  errors. 

For  values  of  (2e+2)  > 100  that  are  not  Included  in  the  table  of 
Appendix  B,  an  approximation  Is  used  where  the  value  of 

X2  . (2e+2)  a 1/2  [1.286  + JlV  -1  ]2 

• y 

where  V ■ (2e+2) 


Using  the  approximation,  a 90%  confidence  level  on  the  upper 
bound  of  the  Type  I error  rate  may  be  determined  for  both  test  pro- 
grams. 


Phase  II,  Period  2: 


Errors  a 1/4  [1.286  + /2(388)-l  ]2  a 212 

w # 7 


Upper  Bound  Error  Rate 


212 

3639 


5.83% 


*See  Appendix  B. 


FI aid  Tests 


ErrorsQ  g » 1/4  [1.286  + /2T938)-!  ]2  - 497 

497 

Upper  Round  Error  Rate  a ■ • ~ - 6.95% 

Thus,  the  best  estimate  of  the  overall  Type  I error  rate  Is  5.30%  and 
6.54%  for  Phase  II  and  the  Field  Teat,  respect l"ely,  with  a 90%  con- 
fidence level  on  the  upper  bound  of  5.83%  for  Phase  II  and  6.95%  for 
the  Field  Test.  This  data  is  presented  in  Table  IV. 


Table  IV 


Confidence  Limits  for  Type  I Error  Rates 


Phase  II 

Field 

Test 

Total  Attemtps 

3639 

7152 

Total  Errors 

193 

468 

Type  I Error  Rate 

5.30% 

6.54% 

Upper  Bound  with  90%  Confidence 

5.83% 

6.95% 

6. 1.2.2  Weekly  Type  I Error  Rates  and  Possible  Causes.  As  people 
used  the  AFV  system,  the  operators  observed  the  verification  process. 
When  u person  had  a type  I error,  a Type  I error  message  was  sent  to 
a teleprinter  and  the  operator  noted  any  circumstances  that  he  had 
observed  that  may  have  contributed  to  the  failure,  e.R.,  used  wrong 
finger.  Improper  use  of  terminal,  etc.  In  addition,  the  operator 
questioned  the  person  that  failed  for  possible  reasons,  e.g. , cold 
hands,  scarred  finger,  etc.  Tables  V and  VI  for  Phase  II  and  the 
Field  Test, respectively, list  the  weekly  number  of  verifications  and 
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Phase  II 


Field  Test  - Weekly  Type  I Error  Data 


failures,  the  reasons  given  or  observed  for  failing,  and  their  fre- 
quency of  occurrence.  It  should  be  noted  that  not  all  those  that 
failed  were  questioned  or  observed,  and  data  is  not  available  as  to 
how  many  successful  verifications  occurred  for  people  with  cold  hands, 
scarred  fingers,  etc.  A discussion  of  each  table  follows: 


TABLE  V.  In  comparing  the  total  verifications  and  failures 
between  Table  III  and  Table  V,  there  is  a discrepancy.  This  dis- 
crepancy arises  because  the  data  for  Table  III  was  taken  from  com- 
puter statistics.  (The  computerized  data  of  Table  III  is  included 
in  parentheses  in  Table  V for  comparison.)  The  data  for  Table  V was 
taken  from  log  sheets  and  daily  statistics  and  it  is  possible  that 
slight  errors  were  made  when  the  data  was  manually  recorded  and  col- 
lated. It  is  felt,  however,  that  the  differences  are  slight  and  there- 
fore of  no  consequence. 

Injured  Finger.  Of  the  260  Type  I errors,  nine  people  re- 
sponded that  they  may  have  failed  because  of  injuries  to  fingers, 
e.g. , cuts  and  scratches,  or  because  of  roughened  hands  acquired 
while  gardening,  doing  cement  work,  etc. 

Wrong  Finger.  Test  participants  were  observed  to  use  wrong 
fingers  for  both  their  Primary  and  Alternate  finger.  In  Phase  II 
this  resulted  in  a single  Type  I error.  However,  there  were  37  known 
cases  where  individuals  were  required  to  use  their  Alternate  finger 
because  they  used  a wrong  finger  for  their  Primary.  As  will  be  dis- 
cussed in  6.2.6,  the  Alternate  finger  has  a lower  probability  of 
passing  verification  than  the  Primary  finger;  therefore,  the  number 
of  Type  I errors  attributable  to  persons  using  a wrong  finger  are 
higher  than  the  single  incident  of  Table  V indicates.  For  the  Field 
Test  (Table  VI),  15  incidences  of  people  using  a wrong  finger  for  both 
their  Primary  and  Alternate  were  noted.  Occurrences  of  people  using 
a wrong  finger  for  Just  their  Primary  were  not  observed  in  the  Field 
Test. 


The  total  contribution  of  Type  I errors  due  to  wrong  finger  was 
difficult  to  identify,  and  it  is  felt  that  there  were  many  undetected 
errors  of  this  nature. 

Consistent  Failures.  Three  people  with  the  highest  Type  I 
error  rate  were  selected  and  the  number  of  times  per  week  that  they 
caused  a Type  I error  was  analyzed.  Choosing  only  the  three  highest 
was  strictly  arbitrary.  They  generated  15%  and  25%  of  the  Type  I 
errors  in  Phase  II  and  the  Field  Test  respectively.  These  people, 
as  did  others,  failed  consistently  even  after  several  reenrollments. 

A photograph  of  a typical  fingerprint  image  belonging  to  a person  who 
failed  often  is  shown  in  Figure  11.  The  fingerprint  image  is  much 
more  complex  than  the  normal  fingerprint  image  shown  in  Figure  12. 

Other.  In  this  column  are  the  number  of  people  who  failed 
and  were  either  not  asked  or  when  they  were  asked,  no  reason  co»>ld 
be  given  for  the  failure.  Included  in  this  group  are  people  who  may 
have  had  damaged  fingers,  or  used  wrong  fingers. 

TABLE  VI.  Again,  in  comparing  the  total  verification  and 
failure  data  of  Table  III  and  Table  VI,  there  are  slight  differences 
due  mainly  from  the  two  ways  in  which  the  data  was  accumulated 
(manually  and  computer).  The  differences  are  slight,  however,  and 
have  no  bearing  on  the  outcome  or  conclusions.  In  addition  to  the 
failure  categories  discussed  for  Phase  II,  there  were  several  addi- 
tional categories  for  the  Field  Test. 

Cold  Hands.  There  had  been  some  concern  prior  to  the  Field 
Test  that  people  with  cold  hands  would  be  responsible  for  higher 
Type  I error  rates.  During  the  Field  Test,  therefore,  when  people 
failed  they  were  asked  if  their  hands  were  cold.  This  was  purely 
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Figure  11.  Fingerprint  Image  of  a Person 
Who  Failed  Consistently. 


Figure  12.  Fingerprint  Image  of  a Person 
Who  Passed  Consistently. 
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a subjective  evaluation,  but  of  the  480  fallees  40  gave  a positive 
response.  The  number  of  people  with  cold  hands  that  passed  the  veri- 
fication could  not  be  determined.  This  number  most  likely  was  quite 
high  because  of  the  time  of  year  in  which  the  Field  Test  was  con- 
ducted and  also  because  when  entering  the  WSA,  the  test  participants 
went  immediately  to  the  AFV  terminal  without  having  a chance  to 
warm  up.  Cold  hands  was  not  a factor  in  Phase  II  because  people 
worked  indoors  and  the  time  of  year  was  late  summer  and  early  fall. 

The  Field  Test  did  experience  higher  Type  I error  rates  than 
Phase  II  and  most  of  the  test  participants  in  the  Field  Test  had 
cold  hands  when  verifying,  which  tends  to  support  the  hypothesis 
that  people  with  cold  hands  will  be  responsible  for  a higher  Type  I 
error  rate. 

Wrong  ID  Number.  Of  the  480  Type  I errors,  13  were  due  to 
Instances  of  test  participants  attempting  to  verify  on  another  per- 
son's reference  file.  This  happened  when  a person  wanting  to  verify 
made  the  mistake  of  entering  another  person's  ID  number. 

It  is  hypothesized  that  the  reason  this  occurred  in  the  Field 
Test  and  not  in  Phase  II  was  because  of  the  difference  in  the  distri- 
bution of  the  ID  numbers.  In  the  Field  Test,  the  ID  number  of  the 
270  enrollees  were  closely  distributed  between  0005  and  0469.  In 
Phase  II,  the  ID  numbers  of  the  200  enrollees  were  much  more  widely 
dispersed  between  0052  and  9500.  Intuitively  one  can  see  that  a 
Field  Test  participant  when  making  a mistake  had  a greater  chance 
of  entering  another's  ID  than  the  Phase  II  participant. 

Type  I errors  due  to  wrong  ID  was  not  unique  to  the  AFV  system 
in  the  Field  Test;  the  other  two  systems  experienced  similar  results. 
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Difficulty  in  Using  Terminal.  There  were  a few  people  who, 
due  to  the  structure  of  their  hand  and/or  fingers,  had  a difficult 
time  in  positioning  their  finger  in  the  finger  scanner.  The  number 
of  people  who  had  difficulty  and  also  caused  a Type  I error  are 
listed.  Since  no  errors  of  this  type  were  evident  in  the  last  four 
weeks  of  test,  it  would  appear  that  the  individuals  were  able  to 
adjust  to  their  particular  problem. 

6.2  OBJECTIVE  24  - TYPE  I ERROR  ANALYSIS  IN  NON-REAL  TIME 

6.2.1  Statement  of  Objective 

To  determine  how  various  parameters  affect  the  Type  I errors 
and  system  performance. 

6.2.2  Type  I Error  Rate  vs  Time  of  Day 

The  Type  I errors  as  a function  of  the  morning  (AM)  and  after- 
noon (PM)  are  listed  in  Table  VII  for  both  males  and  females. 

A 

In  all  instances  the  error  rate  in  the  morning  equals  or  ex- 
ceeds the  error  rate  in  the  afternoon.  To  test  if  the  error  rates 
between  the  morning  and  afternoon  groups  are  significantly  different, 
the  F-ratio  test  is  applied. 

Phase  II,  Period  2,  AM  vs  PMs  T - x - 1.57 

9(130,260)  a 1.20 
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T 3198  300  . 

Field  Test,  AM  vs  PM.  1 " 3954  X 170  “ 

F „ (340,600)  m 1.11 

Ctm.  9 

With  the  results  ol  the  F ratio  test,  one  can  say  with  902  confidence 
that  the  error  rates  between  morning  and  afternoon  are  significantly 
different  for  both  test  programs.  It  has  been  suggested  that  this 
occurs  because  the  hand  Is  relatively  dry  and  oil  free  in  the  morn- 
ing. 

6.2.3  Type  I Error  Rate  vs  Male  and  Female 

The  total  error  rates  experienced  by  males  and  females  for 
Phase  II  and  the  Field  Test  are  presented  in  Table  VIII. 


Table  VIII 

Type  I Error  Rate  vs  Male  and  Female 


Phase  I] 

, Period  2 

Field  Test 

MALE 

FEMALE 

MALE 

FEMALE 

Attempts 

2950 

689 

6669 

483 

Type  I Error 

135 

58 

436 

32 

Error  Rate 

4.582 

8.422 

6.542 

6.632 

Though  the  results  of  the  Field  Test  indicate  a slightly  higher 
Type  I error  rate  for  the  females,  the  results  are  less  significant 
statistically  because  of  the  small  (6)  female  population  using  the 
system. 
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To  determine  if  the  error  rates  between  males  and  females  are 
significantly  different  in  Phase  II,  the  F-ratio  test  is  applied. 

Phase  II,  Male-Female: 


T - 2950  _59 

689  136 


1.86 


F^_  9 (272,118)  « 1.20 


For  Phase  II,  the  F ratio  test  shows  with  90%  confidence 
that  the  error  rates  between  males  and  females  are  significantly 
different. 


An  F-ratio  test  for  the  Field  Test  would  not  be  a good  indicator 
because  of  the  low  female  population. 

It  is  suggested  that  the  reason  females  have  a higher  Type  I 
error  rate  is  because  a female's  fingerprint  when  compared  to  a 
male's  has  a finer  ridge  structure  which  may  approach  or  exceed  the 
resolution  of  the  finger  scanner. 

6.2.4  Type  I Error  Rate  vs  Age  Distribution 

Table  IX  lists  the  Type  I error  rates  as  a function  of  a persons 
age  for  both  test  programs. 

For  Phase  II,  the  verification  attempts  were  predominantly  made 
by  people  between  the  ages  of  26  to  55  while  for  the  Field  Test  the 
predominant  ages  ranged  from  18  to  45. 
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Type  I Error  vs  Age 


One  might  anticipate  that  because  of  dryer  skin,  and/or  more 
complex  fingerprints  due  to  scars,  etc.,  that  the  Type  I error  rate 
would  increase  as  a function  of  age.  Phase  II  shows  this  increasing 
trend  but  in  the  Field  Test  the  younger  population  (<25)  experienced 
the  highest  error  rate  and  there  was  insufficient  data  in  the  older 
categories  to  support  the  hypothesis  that  older  people  would  be  re- 
sponsible for  a higher  Type  I error  rate. 

6.2.5  Type  I Error  Rate  vs  Day  of  the  Week 

The  Type  I error  rate  as  a function  of  the  day  of  the  week  is 
shown  in  Tables  X and  XI  for  the  entire  Phase  II  and  Field  Test, 
respectively.  For  Phase  II,  the  error  rates  appear  variable  as  a 
function  of  the  day  of  the  week  with  no  specific  trend.  For  the 
Field  Test,  the  overall  error  rate  was  lowest  on  Monday  with  a some- 
what stable  error  rate  for  the  middle  of  the  week  with  a sharp  in- 
crease on  Friday.  No  reason  could  be  found  for  the  sharp  increase 
in  error  rate  on  Friday. 

The  data  for  females  is  limited  for  both  tests  but  is  shown  for 
comparison. 


6.2.6  Number  of  Verification  Attenpts  vs  Scan  Number  for  Decision 

In  a six  scan  decision  strategy  each  attempt  is  given  six  scans 
in  which  to  pass:  three  on  the  Primary  finger  and  three  on  the 
Alternate  finger.  The  attempt  is  considered  a success  on  the  first 
scan  that  passes  the  decision  criteria.  An  attempt  generates  a Type  I 
error  when  all  six  scans  fail  to  pass.  The  total  number  of  verifica- 
tion attempts  that  passed  in  each  of  the  six  scans  for  both  test 


Table  X 


Type  1 Error  vs  Day  of  Week  - Phase  II 
(Periods  1 and  2) 


Day  of  Week 

Mon 

Tues 

Wed 

Thurs 

Fri 

Male 

Attempts 

750 

886 

972 

896 

577 

Type  1 Error 

42 

38 

35 

43 

26 

Error  Rate 

(%) 

5.6 

4.3 

3.6 

4.8 

4.5 

Female 

■ 

Attempts 

168 

198 

236 

203 

Type  1 Error 

14 

14 

14 

24 

KSI 

Error  Rate 

(%) 

8.3 

7.1 

5.9 

11.8 

Total 

■ 

Attemp  ts 

919 

B 

1211 

1090 

735 

Type  1 Error 

56 

mm 

59 

67 

37 

Error  Rate 

(%) 

6.1 

■ ■ ■ 

4.9 

6.14 

5.0 

‘ 
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phases  is  presented  in  Table  XII.  Both  tests  had  similar  results 
with  about  86%  of  the  total  attempts  passing  on  the  Primary  finger 
and  about  8%  of  the  total  attempts  passing  on  the  Alternate  finger. 

Of  those  attempts  requiring  the  Alternate  finger,  60%  and  54% 
passed  the  decision  criteria  for  Phase  II  and  the  Field  Test  re- 
aped tvely. 

The  data  from  Table  XII  can  be  plotted  to  show  the  probability  of 
passing  verification  for  each  scan  of  a six  scan  decision  strategy. 

The  probability  data  is  obtained  for  each  scan  by  subtracting  from 
the  total  number  of  attempts  all  previous  successful  scan  attempts 
and  dividing  the  remaining  number  into  the  number  of  successful 
attempts  for  the  scan  of  interest. 

The  probability  of  passing  verification  for  each  of  six  scans  is 
plocted  in  Figure  13.  The  plot  is  representative  of  both  test  pro- 
grams and  shows  that  the  Type  I error  rate  is  dependent  on  scan 
number  and  that  an  individual  requiring  the  use  of  the  Alternate 
finger  has  a lower  probability  of  passing  verification. 

6.2.7  Type  I Error  Rate  Vs  Primary  and  Secondary  Matcher 

As  discussed  in  4.2,  one  of  two  algorithms  la  assigned  to  an 
individual  for  matching  fingerprint  data  with  data  in  the  reference 
file.  The  algorithm  assigned  most  frequently  is  the  Primary  matcher 
with  relatively  few  individuals  requiring  the  Secondary  matcher. 

The  number  of  individuals  assigned  the  Primary  and  Secondary  matcher 
and  the  associated  Type  I errors  generated  is  detailed  in  Table  XIII 
for  a three  scan  decision  strategy.  A three  scan  decision  strategy 
is  used  so  that  the  error  rates  for  the  Primary  and  Alternate  finger 
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Table  XII 


Verification  Attempts  Passed  vs  Scan  Number 


Attempts  Passed 

% of  Total 

Passing 

Scan 

Phase  II 

Field  Test 

Phase  II 

Field  Test 

fl 

2485 

4789 

68.28 

66.96 

PRI- 

MARY 

2 

507 

998 

13.93 

13.95 

3 

156 

361 

4.29 

5.04 

4 

184 

342 

5.05 

4.78 

AL- 

TER- 

5 

69 

119 

1.90 

1.66 

NATE 

6 

45 

75 

1.24 

1.04 

Fai led 

193 

468 

5.30 

6.54 

Totals 

3639 

7152 

Figure  13.  Probability  of  Verification  vs  Scan  Number. 


Table  XIII 


can  be  determined  separately.  In  a three  scan  decision  strategy, 

A Type  I error  is  generated  when  an  attempted  verification  fails  on 
only  three  scans  as  compared  to  six  scans  for  a six  scan  decision 
strategy.  The  result  of  a three  scan  strategy,  because  there  are 
fewer  chances  to  pass,  is  a higher  Type  1 error  rate.  The  three 
scan  decision  strategy  Type  I data  was  generated  by  computer  analysis 
of  Phase  II  and  the  Field  Test  data. 


In  Phase  II  there  were  22  individuals  or  11.17%  of  the  popula- 
tion that  were  assigned  the  Secondary  matcher  for  either  one  or 
both  of  their  Primary  and  Alternate  flnger(s).  In  the  Field  Test, 
there  were  19  or  9.13%  of  the  population  that  were  assigned  the 
Secondary  matcher.  In  both  Phase  II  and  the  Field  Test,  for  a three 
scan  strategy,  individuals  assigned  the  Secondary  matcher  were 
responsible  for  a considerably  higher  Type  I error  rate.  To  test 
if  the  Type  I error  rates  between  the  Primary  and  Secondary  matcher 
are  significantly  different,  the  F-ratio  test  is  applied. 


Phase  II: 


_ 4424  91 

T " 234  X 671 


- 2.56 


F „ (1342,  182)  as  1.15 
a-. 9 

Field  Test: 


_ 7993  418 

T ■ ■ ■ — v ■- 

716  1156 

F (2312,  836) 

d*.? 


4.04 


1.07 
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Thus  one  can  say  with  90%  confidence  that  the  Primary  and 
Secondary  matchers  have  significantly  different  Type  I error  rates. 

6.3  OBJECTIVE  25  - INDEPENDENCE  OF  TYPE  I SCORES 


6.3.1  Statement  of  Objective 

To  determine  the  independence  of  Type  I errors  for  repeated  uses 
of  the  system  by  individual  enrollees  as  well  as  when  compared  against 
other  enrollees. 

6.3.2  Distributl  of  Type  I Errors 

From  the  real  time  data,  the  number  of  Type  I errors  and  the 
frequency  at  which  test  participants  made  the  errors  were  obtained. 
Tables  XIV  and  XV  present  this  data  for  both  Phase  II  and  the  Field 
Test  respectively. 

Column  1 lists  in  consecutive  order  the  number  of  Type  I errors 
and  column  2 lists  the  number  of  people  responsible  for  that  number 
of  Type  I errors.  The  total  number  of  errors  made  by  each  group  is 
listed  in  column  3 and  is  obtained  by  multiplying  the  numbers  across 
columns  1 and  2;  e.g. , from  Table  XIV,  28  people  out  of  the  total  par- 
ticipating population  in  Phase  II  each  had  a single  Type  I error  for 
a total  of  28  errors.  The  group  of  people  that  had  no  Type  I errors 
include  only  those  that  verified  after  being  enrolled.  Those  that 
were  enrolled  and  never  verified  are  not  included. 

From  these  tables,  the  data  for  plotting  a distribution  of  errors 
is  obtained.  Figure  14  shows  this  distribution  with  about  4.5%  of 
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the  participating  populations  in  Phase  II  and  the  Field  Test  re- 
sponsible for  about  50%  of  the  Type  I errors.  The  data  shows  that 
the  Type  I errors  are  heavily  dependent  on  particular  users  and  that 
the  errors  are  not  evenly  distributed  among  the  user  population. 

Of  the  total  test  population,  approximately  69%  and  64%  of  the  Phase 
II  and  Field  Test  respectively  had  no  trouble  passing  the  verification 
process.  It  was  not  uncommon,  however,  for  individuals  to  have  Type  I 
error  rates  over  20%  with  several  individuals  approaching  error  rates 
of  60%. 

6.4  OBJECTIVE  26  - TYPE  II  ERROR  ANALYSIS  IN  REAL  TIME 

6.4.1  Statement  of  Objective 

To  estimate  the  Type  II  error  (erroneously  verifying  identity 
when  the  representation  is  actually  false)  and  to  determine  the  con- 
fidence limits  of  the  Type  II  error  when  imposters  attempt  verifica- 
tion. 

6.4.2  Type  II  Error  Analysis 

During  Phase  II,  various  techniques  were  employed  by  individuals 
to  generate  Type  II  errors.  The  techniques  used  and  the  results  are 
described  in  another  document. 
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6.5  OBJECTIVE  27  - TYPE  II  ERROR  ANALYSIS  IN  NON-REAL  TIME 


6.5.1  Statement  of  Objective 


To  estimate  the  Type  II  error  rate  (erroneously  verifying  iden- 
tity when  the  representation  is  actually  false)  and  to  determine  the 
confidence  limits  of  the  Type  II  error  when  matching  within  the 
enrolled  population  is  performed. 


6.5.2 


II  Error  Rate 


On  every  fourth  verification  attempt  in  Phase  II  and  eighth 
verification  attenpt  in  the  Field  Test,  imposter  data  was  gathered 
for  generating  random  Type  II  error  rates.  The  Imposter  data  con- 
sisted of  3 scans  each  of  the  Primary  and  Alternate  finger  in  a form 
appropriate  for  both  the  Primary  and  Secondary  matchers.  This  data 
was  played  back  during  the  nighttime  hours  against  all  the  reference 
files  in  an  attempt  to  find  a match  between  the  imposter  and  all 
other  enrollees. 


The  Type  II  error  rates  are  based  on  a six  scan  decision  strategy 
just  as  the  Type  I error  rate.  That  is,  each  of  the  three  scans  of 
an  Imposter’s  Primary  and  Alternate  fingers  is  compared  respectively 
against  Primary  and  Alternate  finger  scan  data  in  each  reference 
file,  i.e..  Primary  vs  Primary  and  Alternate  vs  Alternate.  A Type  II 
error  occurred  when  any  one  of  the  six  comparisons  passed  a decision 
threshold.  The  Type  II  error  rate  was  determined  using  the  formula: 


Type  II  Error  Rate  (X)  ■ 


pe  II  Errors 
Attempts 


x 100 


Figure  15  shows  the  daily  Type  II  error  rates  for  Phase  II  and 
the  Field  Test.  There  is  no  indication  of  an  increasing  trend  in 
the  Type  II  error  rate  as  was  evident  in  the  Type  I error  rate  for 
the  Field  Test  (6.1.2). 

For  Phase  II,  an  overall  increase  in  the  Type  II  error  rate 
occurred  in  the  third  week.  This  was  due  to  the  change  made  in  the 
pass/fail  decision  logic  previously  discussed  in  5.2.  Also,  it  may 
be  noticed  that  there  was  no  Type  II  data  generated  for  the  last 
week  of  the  Field  Test.  During  that  week,  data  was  not  collected 
so  that  system  operation  and  user  acceptance  could  be  observed 
without  the  necessity  of  detaining  the  users  to  collect  imposter 
data. 


The  Type  II  error  rates  for  Phase  II  Period  2 and  the  Field 
Test  are  presented  in  Table  XVI  and  XVII  respectively.  The  tables 
divide  the  data  into  three  comparisons:  male  vs  male,  male  vs 
female,  and  female  vs  female. 


In  the  male  vs  male  comparison  (i.e.,  male  imposter  data  is 
compared  against  all  male  reference  file  data)  the  Type  II  error 
rate  is  slightly  higher  for  the  Field  Test.  To  determine  if  the 
Type  II  error  rate  is  significantly  different  between  Phase  II  and 
the  Field  Test,  an  F-ratio  test  is  performed. 


T 


48079  1405 

60636  X 1050 


1.06 


F (2100,  2810)  a 1.05 

a=  .9 
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Table  XVI 

Total  Type  II  Error  Rates  - Phase  II 


' 

Attempts 

Errors 

Error  Rate 

Male  vs  Male 

48,079 

1,049 

2.18  X 

Male  vs  Female 

21 ,327 

301 

1.41  % 

Female  vs  Female 

2,279 

75 

3.29  X 

Totals 

71,685 

1,425 

1.99  X 

Table  XVII 

Total  Type  II  Error  Rates  - Field  Test 


Attempts 

Errors 

Error  Rate 

Male  vs  Male 

60,636 

1,404 

2.32  X 

Male  vs  Female 

5,647 

91 

1.61  X 

Female  vs  Female 

99 

2 

2.02  X 

Totals 

66,382 

1,497 

2.25  X 

Thus,  one  can  say  with  90X  confidence  that  the  male  groups  in  the 
Phase  II  and  Field  Test  have  significantly  different  Type  II  error 
rates,"  Tor  comparison,  the  female  data  is  presented  in  the  table; 
however,  it  is  felt  that  there  were  too  few  females  involved  in  the 
Field  Test  to  make  a statistical  comparison. 

6.5.3  Confidence  Limits  of  the  Type  II  Error  Estimate  (Male  Only) 

Using  the  figures  in  Tables  XVI  and  XVII,  a 90X  confidence  level 
on  the  upper  bound  of  the  Type  II  error  rate  may  be  determined  for 
both  test  programs. 

Phase  II,  Period  2: 

Errors  a 1/4  [1.286  + 

Upper  Bound  Error  Rate 

Field  Test: 


v/2(2100)-l  ]2  = 1092 


1092 

48,079 


- 2.27X 


Errors  a 1/4  [1.286  + v/2(2810)-l]2  - 1454 

w • / 

1454 

Upper  Bound  Error  Rate  w 60  636*  * 2.40X 

Thus,  the  best  estimate  for  the  overall  Type  II  error  rate  for 
males  is  2.18X  and  2.32%  for  Phase  II  and  the  Field  Test  respectively 
with  a 90X  confidence  level  on  the  upper  bound  of  2.27X  for  Phase  II 
and  2.40X  for  the  Field  Test.  This  data  is  presented  in  Table  XVIII. 
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Table  XVIII 


Confidence  Limits  for  the  Type  II  Error  Rate  (Male  Only) 


Phase  II 

Field  Test 

Attempts 

48,079 

60,636 

Errors 

1,049 

1,404 

Error  Rate 

2.18% 

2.32% 

Upper  bound  with  90%  confidence 

2.27% 

2.40% 

6.5.4  Type  II  Error  Rate  Vs  Primary  and  Secondary  Matcher 


As  previously  discussed  In  6.2.7  approximately  11%  and  9%  of 
the  participants  in  the  Phase  II  and  Field  Test  were  assigned  the 
Secondary  matcher.  Table  XIX  presents  the  data  for  comparing  the 
performance  of  the  Primary  and  Secondary  matcher  for  Type  II  errors 
for  a three  scan  decision  strategy. 


In  both  Phase  II  and  the  Field  Test  those  individuals  assigned 
the  Secondary  matcher  were  responsible  for  a lower  Type  II  error 
rate.  To  test  if  the  Type  II  error  rates  between  the  Primary  and 
Secondary  matcher  are  significantly  different  the  F-ratio  test  is 
applied. 


Phase  II: 


T 


6620  2196 

193,743  39 


1.92 


Fa«.9(78»A392)  * 1,21 
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Type  II  Error  Rate  vs  Primary  and  Secondary  Matcher 
(Three  Scan  Decision  Strategy) 


Field  Test: 


T 


12868  2772 

230,108  X 30 


F (60,5544)  « 1.25 

a».9 


Thus , one  can  say  with  90%  confidence  that  the  Primary  and  Secondary 
matcher  have  significantly  different  Type  II  error  rates. 


6.5.5  Type  II  Error  Rate  vs  Imposter  Finger 

In  deriving  the  Type  II  errors  for  Phase  II  and  the  Field  Test, 
all  fingerprint  data  in  the  reference  files  are  scanned  using  a six 
scan  decision  strategy  in  an  attempt  to  find  a match  with  the  imposter 
fingerprint.  Thus,  if  an  imposter's  Primary  finger  is  the  left  index, 
that  finger  was  matched  to  all  Primary  fingers  in  the  reference  file, 
i.e.,  right  and  left  index,  middle,  ring,  and  little  finger. 

Data,  derived  from  computer  analysis,  is  presented  in  Table  XX 
that  shows  the  Type  II  error  rates  for  the  Primary  and  Alternate 
finger  as  a function  of  comparing  an  Imposter  finger  with  the  same 
Primary  or  Alternate  finger  of  a victim  (right  index  against  right 
index,  left  middle  against  left  middle,  etc.).  The  data  in  Table  XX 
are  for  a three  scan  decision  strategy  and  only  male-male  and  female- 
female  comparisons  are  made.  An  approximation  can  be  made  for  the 
Type  II  error  rate  that  could  be  expected  in  a six  scan  decision 
strategy  by  summing  the  error  rates  for  the  Primary  and  Alternate 
finger.  These  totals  are  Included  in  Table  XX.  In  this  analysis, 
the  female-female  comparisons  had  little  effect  on  the  overall  error 
rates  because  of  the  low  female  population. 


71 


X 

X 


T3 

c 

(O 

s- 

s 

m 

MB 

(O 
4-  L. 
O -M 
C/> 
in 

i-  c 
0)  o 
cn«»- 
c to 


(U  ‘f  •*" 


<T5 


La.  U 
0) 

•f-  C 
—I 

u 

t/)  co 

> 

0) 

32! 

(O  -C 

q:  h* 

i. 

£ 

i. 


<D 

Q. 

.>> 


The  data  shows  (e.g. , 3.82%  vs  2.23%  and  5.05%  vs  2.32%)  that 
an  imposter  can  improve  his  chances  of  success  by  knowing  which 
fingers  have  been  assigned  as  the  Primary  and  Alternate  finger  of 
his  victim. 

6.6  OBJECTIVE  28  - INDEPENDENCE  OF  TYPE  II  SCORES 

6.6.1  Statement  of  Objective 

To  determine  the  independence  of  the  Type  II  scores  for  repeated 
uses  of  the  system  by  individual  enrollees  as  well  as  when  compared 
against  other  enrollees. 

6.6.2  Distribution  of  Type  II  Errors 

The  distribution  of  Type  II  errors  among  the  test  populations 
is  obtained  in  the  same  way  as  for  the  distribution  of  Type  I errors 
described  in  6.3.2.  Tables  XXI  and  Table  XXII  present  the  distribu- 
tion data  for  the  Phase  II  and  Field  Test  respectively.  Figure  16 
plots  the  distribution  of  Type  II  errors  for  both  test  phases  and 
shows  that  the  distribution  of  Type  II  errors  is  not  uniform  across 
the  test  population  with  about  14%  of  the  population  responsible  for 
50%  of  the  Type  II  errors.  In  addition,  12.5%  of  the  Phase  II  popu- 
lation and  21.43%  of  the  Field  Test  population  had  no  Type  II  errors. 
The  Type  II  errors  are  dependent  on  particular  users  and  are  not 
evenly  distributed  among  the  population.  It  was  not  unusual  for 
individuals  to  be  responsible  for  Type  II  error  rates  exceeding  10%. 
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Table  XXI 


7 


Distribution  of  Type  II  Errors  - Phase  II  Period  2 


Number  of  Type  II 
Errors 


0 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 
26 
27 

31 

32 

33 

38 

39 
86 


Totals 


Number  of  People 
Responsible 

Total  Type  II 
Errors 

25 

0 

19 

19 

24 

48 

23 

69 

16 

64 

9 

45 

60 

10 

70 

4 

32 

5 

45 

11 

110 

5 

55 

3 

36 

4 

52 

3 

42 

5 

75 

4 

64 

3 

51 

3 

54 

1 

19 

3 

60 

1 

21 

1 

22 

1 

26 

1 

27 

1 

31 

1 

32 

1 

33 

1 

38 

1 

39 

1 

86 

200 

1425 
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Table  XXI 1 


Distribution  of  Type  II  Errors  - Field  Test 


Number  of 

Type  II  Errors 

Number  of  People 
Responsible 

Total  Type  II 
Errors 

0 

57 

0 

1 

28 

28 

2 

33 

66 

3 

23 

69 

4 

16 

64 

5 

16 

80 

6 

18 

108 

7 

9 

63 

8 

5 

40 

9 

9 

81 

10 

7 

70 

11 

5 

55 

12 

5 

60 

13 

8 

104 

14 

4 

76 

15 

4 

60 

16 

1 

16 

17 

2 

34 

18 

1 

18 

20 

3 

60 

21 

1 

21 

22 

1 

22 

23 

1 

23 

24 

1 

24 

25 

1 

25 

26 

1 

26 

28 

1 

28 

30 

1 

30 

32 

1 

32 

36 

1 

36 

37 

1 

37 

41 

1 

41 

Totals 

266 

1497 

F 
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6.7  OBJECTIVE  29  - SENSITIVITY  ANALYSIS  OF  TYPE  I AND  TYPE  II  ERRORS 

TO  THE  DECISION  THRESHOLD 

6.7.1  Statement  of  Objective 

To  determine  the  sensitivity  of  Type  I and  Type  II  error  varia- 
tions to  changing  thresholds. 

6.7.2  Sensitivity  Analysis 

For  a Type  I error  to  occur,  the  score  obtained  from  matching 
an  individual’s  fingerprint  with  the  same  individual's  data  in  the 
reference  file  must  be  less  than  the  pass/fail  decision  threshold. 

For  a Type  II  error  to  occur,  the  score  obtained  from  matching  an 
impostor's  fingerprint  data  with  that  of  his  victim's  must  be  equal 
to  or  greater  than  the  pass/fail  decision  threshold.  There  is  a 
trade-off,  therefore,  between  Type  I and  Type  II  errors  and  the  level 
of  the  pass/fail  decision  threshold:  as  the  threshold  is  raised, 
the  system  will  experience  more  Type  I errors  at  the  expense  of  fewer 
Type  II  errors  and  vice  versa. 

To  evaluate  the  sensitivity  of  Type  I and  Type  II  errors,  the 
Field  Test  data  was  collated  to  show  the  percentage  of  errors  occurring 
for  the  Primary  matcher  on  the  first,  second  and  third  scans  as  a func- 
tion of  the  threshold  level.  Figure  17  shows  the  data  plotted  for 
each  of  the  first  three  scans.  Data  for  Phase  II  is  not  shown  because 
It  is  similar  to  the  Field  Test  data.  When  an  individual's  score 
falls  below  the  decision  threshold  (score  minus  the  threshold  is 
negative),  a Type  I error  is  generated.  When  an  impostor's  score  is 
equal  to  or  above  the  decision  threshold  (score  minus  the  threshold 
Is  zero  or  positive),  a Type  II  error  is  generated.  The  Type  I error 
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rate  is  shown  to  be  dependent  on  the  scan  number,  e.g.,  from  Figure 
17  at  a score  minus  threshold  = 0 there  is  a 33%  Type  1 error  rate 
for  scan  1,  a 58%  Type  I error  rate  for  scan  2 and  a 74%  Type  I error 
rate  for  scan  3.  The  Type  II  error  rate  of  Figure  17  is  shown  to  be 
independent  of  scan  number,  e.g.,  the  Type  II  error  rate  is  approxi- 
mately the  same  for  the  first,  second,  and  third  scans. 

The  data  in  Figure  17  can  be  configured  to  show  the  overall 
sensitivity  of  the  Type  I and  Type  II  error  rate  to  threshold  for  a 
three  scan  decision  strategy.  In  a three-scan  decision  strategy,  a 
Type  I error  is  generated  when  an  individual  fails  all  three  scans. 
Therefore,  to  obtain  Type  I error  data  for  a three-scan  decision 
strategy,  the  percentages  in  each  of  three  scans  as  a function  of 
score  minus  threshold  are  multiplied  together;  e.g.,  for  the  Field 
Test,  at  a score  minus  threshold  = 0,  33%  for  scan  1 x 58%  for  scan 
2 x 74%  for  scan  3 = 14.16%  error  rate  for  a three  scan  decision 
strategy. 

A type  II  error  is  generated  in  a three-scan  decision  strategy 
when  an  imposter  matches  a reference  file  on  any  one  of  the  three 
scans.  Therefore,  to  obtain  a type  II  error  curve  showing  the  per- 
formance of  a three-scan  decision  strategy,  the  percentage  of  people 
in  each  scan  as  a function  of  score  minus  threshold  is  approximated 
by  summing  the  errors  in  each  scan,  e.g.,  for  the  Field  Test,  a 
score  minus  threshold  = 0,  .6%  for  scan  1 + .6%  for  scan  2 + .6%  for 
scan  3 = 1.8%  error  rate  for  a three-scan  decision  strategy. 

The  results  of  a three-scan  decision  strategy  and  the  sensitivity 
of  the  error  rates  to  threshold  for  the  Primary  matcher  are  shown  in 
Figure  18  for  the  Field  Test.  The  Phase  II  results  would  be  similar. 
Ideally,  it  would  have  been  preferred  to  show  the  same  type  of 
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Error  Rate  * 


, 

Kl*ure  18  Sensitivity  of  Type  I and  Type  II  Errors  to  Threshold 
(Three  Scan  Strategy  - Field  Test) 


analysis  for  the  six-scan  decision  strategy  used  in  Phase  II  and  the 
Field  Test  and  included  the  data  for  the  Secondary  matcher  for  both 
Primary  and  Alternate  fingers,  but  data  was  not  available  to  do  this. 
Had  data  been  available,  an  analysis  similar  to  the  one  just  discussed 
for  the  Primary  matcher  could  have  been  performed  for  the  Secondary 
matcher.  The  data  could  then  have  been  combined  to  generate  Type  I 
and  Type  II  error  curves  for  a six-scan  decision  strategy  employing 
both  Primary  and  Secondary  matchers.  A complete  analysis  such  as  this 
would  have  shown  the  Type  I error  curve  of  Figure  18  intersecting 
the  ordinate  at  6.54%  (Section  6.1.2)  and  the  Type  II  error  curve 
intersecting  the  ordinate  at  2.32%  (Section  6.5.2).  It  is  evident  that 
if  one  elected  to  lower  the  Type  I error  rate  to  better  meet  BISS 
requirements  by  lowering  the  threshold  (i.e.,  in  Figure  18  causing  the 
Type  I and  Type  II  error  curves  to  move  to  the  right),  the  Type  II 
error  rate  would  increase  rapidly  and  further  exceed  the  BISS  require- 
| ment  of  2%. 

6.8  OBJECTIVE  30  - VERIFICATION  TIME  ANALYSIS 

6.8.1  Statement  of  Objective 

To  determine  the  average  time  required  for  verification  and  the 
variance  about  this  time. 

6.8.2  Verification  Time 

Table  XXIII  lists  the  parameters  for  calculating  the  time  from 
the  moment  an  ID  number  is  accepted  until  a THANK  YOU  or  TRY  FINGER 
AGAIN  message  is  displayed.  The  data  does  not  include  the  time  re- 
quired to  process  imposter  data  nor  does  it  include  the  time  required 
for  the  users  to  enter  their  identification  numbers. 


81 


Table  XXIII 


Parameters  for  Determining  Verification  Time 


PHASE  II 

FIELD  TEST 

Total  No.  of  Scans  (St) 

9,019 

13,089 

Total  Attempts  (At) 

5,039 

7,152 

Total  Time 

34,951  sec. 

52,100  sec. 

Avg.  Time/Scan  (t) 

3.88  sec. 

3.98  sec. 

Standard  Deviation 

0.9  sec. 

1.06  sec. 

To  obtain  the  average  time  for  a pass/fail  decision  to  be  made, 
the  average  time  per  scan  is  multiplied  by  the  number  of  scans  re- 
quired for  a decision.  Also  to  be  factored  into  the  time  element, 
is  the  time  it  takes  for  an  user,  after  the  first  scan,  to  remove 
and  reinsert  his  finger  and  press  the  ENTER  button  on  the  keypad. 

This  time  has  been  estimated  to  take  2 seconds.  With  the  above  in- 
formation the  following  formula  is  used  to  determine  the  verification 
time: 


VT  - 2(N-1)  + (N  x t) 
where 

VT  - Verification  Time 

St 

N ■ Average  Number  of  scans  per  attempt  ■ - — 

At 

t " Average  time  per  scan 


i 


' 
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For  Phase  II: 

VT  - 2(1.79  - 1)  + (1.79  x 3.88)  - 8.53  seconds 
and  for  the  Field  Test: 


Vm  - 2(1.83  - 1)  + (1.83  x 3.98)  - 8.94  seconds 


7.0  CONCLUSIONS 


. 

ii 


i 


From  the  results  reported  herein  on  both  Phase  II  and  the  Field 
Test,  the  following  conclusions  are  made: 

SUPPORTING 

CONCLUSIONS  AS  SUPPORTED  BY  BOTH  TEST  PHASES  PARAGRAPHS 

1.  The  AFV  Type  I error  rate: 


a.  Does  not  meet  the  BISS  requirement  of  1%  6.1 

b.  Is  significantly  higher  in  the  morning  than  6.2.2 

in  the  afternoon 

c.  For  the  Primary  finger,  was  much  better  6.2.6 

(K14%  Phase  II  and  Field  Test)  than  it 

was  for  the  Alternate  finger  («40%  Phase  II 
and  «46 % Field  Test). 

d.  Is  dependent  on  scan  number  6.2.6 

e.  For  the  Primary  matcher  was  much  better  6.2.7 

(«15%  Phase  II  and  Field  Test)  than  it  was 

for  the  Secondary  matcher  («38%  Phase  II 
and  3:58%  Field  Test) 

f.  Is  not  evenly  distributed  among  the  user  6.3 

population  with  approximately  5%  of  all  users 
generating  50%  of  the  Type  I errors  and  with 
approximately  22%  of  the  users  having  a Type  I 
error  rate  exceeding  5%. 


2.  The  AFV  Type  II  error  rate: 


a.  Does  not  meet  the  BISS  requirement  of  2%  6.5.2 

b.  Does  not  show  an  increase  with  time  since  6.5.2 

enrollment 

c.  For  the  Primary  matcher  was  significantly  6.5.4 

higher  (1.13%  for  Phase  II  and  1.20%  for 

the  Field  Test)  than  it  was  for  the 
Secondary  matcher  (.57%  for  Phase  II  and 
.23%  for  the  Field  Teat). 
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CONCLUSIONS  AS  SUPPORTED  BY  BOTH  TEST  PHASES 

d.  Will  be  higher  for  an  imposter  who  knows 
the  Primary  and  Alternate  finger  of  his 
victim. 

e.  Is  not  evenly  distributed  among  the  user 
population  with  approximately  14%  of  the 
population  generating  50%  of  the  Type  II 
errors. 

f.  Is  independent  of  scan  number 

3.  The  average  verification  time,  excluding  ID 
number  entry  and  finger  repositioning,  is 
approximately  8.7  seconds. 


SUPPORTING 

PARAGRAPHS 

6.5.5 


6.6 


6.7 


6.8 


In  addition,  the  Field  Test  data  (6.1.2)  shows  that  the  Type  I 
error  rate  Increases  with  time.  Initially,  the  Type  I error  rate 
was  about  4%  but  by  the  end  of  the  Field  Test  it  was  close  to  8%. 
Increasing  error  rates  with  time  since  enrollment  were  not  observed 
to  rise  as  rapidly  in  Phase  II  perhaps  because  (1)  there  was  a 
parameter  change  part  way  into  the  test,  (2)  there  were  more  re- 
enrollments performed  in  Phase  II  than  in  the  Field  Test,  and  (3)  the 
Field  Test  was  4-1/2  weeks  longer  in  duration  than  Phase  II  Period  2. 

The  Phase  II  test  data  suggests  that  the  Type  I and  Type  II 
error  rates  are  higher  for  females  than  for  males.  Because  of  the 
limited  number  of  females  in  the  Field  Test,  a comparison  could  not 
be  made. 

It  has  been  hypothesized  that  persons  with  cold  hands  will 
experience  a higher  Type  I error  rate.  Because  most  test  partici- 
pants in  the  Field  Test  had  cold  hands  and  the  Field  Test  did  have 
higher  Type  I error  rates,  the  hypothesis  has  not  been  disproved. 
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Finally,  the  personnel  statistics  of  age,  height,  birth  place, 
and  area  of  primary  education  did  not  exhibit  any  consistent  In- 
fluence on  the  error  rates. 


1 
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APPENDIX  A 


DETERMINING  IF  THE  ERROR  RATES  FROM 
TWO  GROUPS  ARE  SIGNIFICANTLY  DIFFERENT 


In  BISS  data  analysis  it  is  often  desired  to  determine  if  the 
error  rates  from  two  groups  are  significantly  different.  The  BISS 
data  provides  only  samples  from  different  groups.  The  F test  pro- 
vides a test  to  determine  statistically  if  the  underlying  populations 
of  the  different  groups  do  not  have  the  same  error  rates. 

The  binominal  distribution 


PN(k) 


N!  k 

k! (N  - k)!  p 


(1  - P)N"k 


(1) 


is  the  probability  of  having  k errors  in  N trials  where  the  proba- 
bility of  error  is  p and  that  of  no  error  is  1 - p.  Each  trial  is 
taken  to  be  Independent.  In  the  limiting  case  where 

p is  small  (p  « 1) 

N is  large  (N  » 1)  and 

N » k,  (2) 


j 


j 


i 
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the  blncmial  distribution  may  be  approximated  by  the  Poisson  dis- 


tribution as 


(1) 


(3) 


From  the  relationship 

T (x  + 1)  ■ x! 

equation  (3)  may  be  rewritten  as 


P (Np)  - ANPJ 1 

V * r (k  + 1) 


(4) 


The  continuous  variable  Np  is  a Gamma  variate  with  parameter  k + 1 


and  its  distribution,  equation  (4),  is  a Ganna  distribution. 


(2) 


The  probability  that  the  variable  Np  is  in  the  interval  dNp  is 


dP  . 


(k+l)-l  -Np 


<k  + 1) 


dNp 


(5) 


Substituting 


k + 1 - 


(6) 
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into  equation  (5)  yields: 


2l5(n-2)  (Np)ls(n-2)  e~NP 

dp-2 5 d(2«P/2) 

2%(i>-2)  r(n/2) 


- 12N£l2 ® d(2Np) 


2n/2  r(n/2) 


The  probability  density  function 


Pc(2Np)  - 


2n/2  r (n/2) 

is  the  chi-square  distribution  for  the  variate  2Np  with  n degrees  of 


freedom. 


Consider  that  2N1p1  and  2N2p2  are  independent,  and  that  each 
is  chi-square  distributed  with  and  n2  degrees  of  freedom,  re- 
spectively. Then 


T(tl1,n2)  - 


2NlPl  / 

2N2p2  / 


is  the  F variate  and  its  distribution  is  the  F distribution. 
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Replacing  n with  2(k  + 1)  from  equation  (6) , equation  (8)  becomes 


T[2(k1  + 1),  2(k2  + 1)]  - 


2NiPl(2k2  + 2) 
2N2p2(2kL  + 2) 


(9) 


- Vl(k2  + l) 

’ N2p2(kl  + U 

If  the  Error  rates  and  p2  of  the  underlying  populations  are 
hypothesized  to  be  equal,  then  equation  (9)  becomes 


T(2k1  +2,  2k2  + 2) 


NL(k2  + 1) 


N2(kx  + I) 

This  hypothesis  can  be  tested  by  seeing  how  often  It  will  happen  that 
the  F variate  with  parameters  2k^  + 2 and  2k2  + 2,  F(2k^  + 2, 

2k2  4-2),  will  take  on  values?(N^k2  + N^)/(N2k^  + N2>. 

There  are  tables  of  the  cumulative  F distribution  tabulating 
against  a^,  the  values  of  F(a^,a2)  such  that 


_pp  Fp 

- j f[F(alfa2)]  dF  - f f[F(2kx  + 2,  2k£  + 2)]  dF 

•'Q  J(\ 


(10) 


where  f(F)  Is  the  probability  density  function.  From  equation  (10) 

It  la  meant  that  100P  X of  the  time  the  values  of  F(2k^  + 2,  2k2  + 2) 
will  be  < Fp.  Thus,  If  the  observed  T is  less  than  or  equal  to  Fp, 
then  there  la  a 100P  Z "level  of  confidence"  and  T - (N^k2  + N^)/ 
(N^i  + N2)  is  an  F variate  and  therefore  the  underlying  hypo- 
thesis of  equal  error  rates  cannot  be  rejected. 

If  P is  chosen  large  enough,  then  in  the  case  of  T > F^  it  can 
be  concluded  that  it  is  very  unlikely  that  T is  F distributed.  The 
hypothesis  of  equal  error  rates  can  then  be  rejected.  The  inverse 
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reasoning  is  not  possible.  If  T <_  Fp,  one  cannot  say  that  T is  F 


distributed  and  the  hypothesis  is  true,  but  only  that  the  hypothesis 


is  not  inconsistent  with  the  results. 


For  example,  if  are  the  entry  attempts  of  group  i and 


are  the  errors  in  group  1,  then  for 


! a 


H - 1646 


N2  - 1027 


T[2(23  + 1),  2(19  + 1)]  - (23  + 1)' 


T(48,  40)  - 1,336 


From  the  F tables  for  P ■ .9,  the  value  of  F such  that 
P P 


FP 


f[F(48,  40)]  dF 


F - 1.49 
P 


For  this  example,  T2sF  Q and  therefore,  at  the  90Z  confidence 

P m 

level  the  hypothesis  that  the  two  groups  have  equal  error  rates 
cannot  be  rejected. 
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In  this  manner,  the  F test  can  be  used  to  determine  If  the 
error  rates  of  two  different  groups  are  significantly  different  In 
a statistical  sense. 
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APPENDIX  B 


COMPUTING  AN  UPPER  BOUND  ON  ERRORS 


The  basis  for  the  formula  to  compute  error  bounds  in  identity 
verfication  is  described  below.  Assume  that  the  number  of  verifi- 
cation errors  for  a given  number  of  (trials)  entry  attempts  is  statis- 
tical equivalent  to  the  number  of  failures  of  a piece  of  equipment 
for  a given  number  of  operating  hours.  Specifically,  failures  are 
replaced  with  errors  and  operation  hours  are  replaced  with  entry 
attempts.  Now,  the  upper  bound  on  the  number  of  errors  at  a given 
confidence  level  can  be  estimated  for  a fixed  number  of  trials  in 
the  same  way  that  the  number  of  failures,  at  a given  confidence  level, 
can  be  estimated  for  a fixed  number  of  operating  hours. 

The  formula  used  to  estimate  the  lower  bound  of  the  mean  time 

(A) 

between  failures  (MTBF)  for  a fixed  time  test  is  given  by 
MTBFc  = 2T  / [x^_c  (2f  +2)]. 

where 

T is  time  period 

c is  confidence  level,  e.g. , 90% 

f is  number  of  failures 

2f  + 2 number  of  degrees  of  freedom 

The  upper  bound  on  failure  rate  is  the  inverse  of  the  lower  bound 
on  MTBF.  Thus 
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Upper  Bound  Failure  Rate  » 1/MTBF  - X (2f  + 2) /2T 

lover 

The  upper  bound  on  the  number  of  failures  divided  by  the  time  period 
T Is  the  upper  bound  on  the  failure  rate 

Failures  ■ x?  (2f  + 2)/2. 
c 1-c 

Substituting  errors,  e for  failures,  f yields 
Errors^  -[x^  <2e  + 2^/2. 

An  example  of  the  use  of  this  equation  follows.  On  the  first 
entry  attempt  using  the  speaker  verification  system  there  were  8 
failures  to  verify  an  authorized  Individual  out  of  193  entry  attempts 
leading  to  an  e ■ 8.  The  upper  bound  on  the  error  at  90%  confidence 
Is  given  by  (see  Reference  5 for  chi-square  distribution  tables) 

Errors  9 »A'2(18)/2  - 26/2  - 13. 

We  can  say  with  90%  confidence  that  no  more  than  13  errors  out  of  193 
trials  will  occur  on  the  first  trial. 

The  best  estimate  of  the  error  rate  is  4.14%  and  a 90%  confid- 
ence level  on  the  upper  bound  of  the  error  rate  Is  4.14%  x 13/8  - 6.73%. 
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